.TH COLLECTL 1 "APRIL 2003" LOCAL "Collectl" -*- nroff -*-
.SH NAME

.B collectl
- Collects data that describes the current system status.

.SH SYNOPSIS
Record Mode - read data from live system and write to file or display on terminal

.B collectl [-f file] [options]

Playback Mode - read data from one or more raw data files and display
on terminal

.B collectl -p file1 [file2 ...] [options]

.SH OPTIONS

Record Mode

In this mode data is taken from a 
.BR live
system and either displayed on the
terminal or written to one or more files or a socket.

.B "--align"
.RS
If the HiRes modules is present, 
.BR collectl
sample monitoring will be aligned such that a sample will always be taken at the 
top of a minute (this does NOT mean the first sample will occur then) so that all
instances of collectl running on any systems which have their clocks synchronized 
will all take samples at the same time.  Furthermore, if one is doing process 
monitoring, those samples will also be taken at the top of the minute and so can 
delay the start of sampling up to 2 full process monitoring intervals.
.RE

.B "--all"
.RS
Collect summary data for ALL subsystems except slabs, since slab monitoring requires
a different monitoring interval.  You can use this switch anywhere -s can be used
but not both together.  If the system supports lustre and/or interconnect monitoring
those statistics will be provided but the warnings produced when they are not 
available you try to select them with -s will not be displayed.
.RE

.B "-A, --address address[:port] | server[:port]"
.RS
In the first form, one species an address or hostname and optional port.
All data is then written to that socket prefaced with the current host
name at the named address and port until
the socket is closed, at which time collectl will exit.

In the second form one enters the text 'server' and optional port.  
In this form, collectl runs as a server, waiting for a connection and 
once established writes data on that socket.  The key difference here is
if the client exists collectl keeps running and will again look for a
new connection, allowing it to survive client restarts or crashes.
If collectl receives the text 'exit' from the client, it will shut down.

The default port is set at 1234 but can be changed - see collectl.conf.  

In both forms, one can additionally request local data logging by 
specifying a combination of -P and -f.  See
.B "man collectl-logging"
for more details.
.RE

.B "-c, --count Samples"
.RS
The number of samples to record. This is one way of 3 ways of describing
how long collectl should run (see
.BR -r
and
.BR -R
).  Note that these 3 switches are mutually exclusive.
.RE

.B "-C, --config filename"
.RS
Name/location of the collectl configuration file.
If not specified, 
.BR collectl
searches for
.BR collectl.conf
first in /etc (the default), then in the same directory the
.BR collectl
executable is in, and finally the current working directory.
.RE

.B "-D, --daemon"
.RS
Run
.BR collectl
as a daemon, primarily used when starting as a service.  One
caveat about this mode is you can only run one copy.
.RE

.B "--expdir directory"
.RS
This names the directory to write exported output to when writing
to a local file, such as lexpr/sexpr do when then write the L/S files.  
Note that it is up to the script to decide if it chooses to use this 
switch as not all may choose to write to a file.  The default is to
use the directory specified by -f.
.RE

.B "--export file[,options]"
.RS
This requests that collectl does not print anything on the terminal (or
send it to a socket) using the standard brief/verbose/plot formats.
Instead it executes a perl 'require' on the named file, using an extension of 
ph if not specified.  It first looks in the current directory and if not
there the directory the executable is in.  It then calls the
function 'file'Init(options) towards the beginning of collectl and again as 
simply  'file'(@options) to generate the exported formatted output.  See the document
sections on Exporting Custom Output and Logging for more details.
.RE

.B "-f, --filename Filename"
.RS
This is the name of a file to write the output to.  See the
description of
.BR File 
.BR Naming
for further details.
.RE

.B -F, --flush seconds
.RS
Flush output buffers after this number of seconds.  This is equivalent to 
issuing 
.B kill -s USR1
at the same frequency (but a lot easier!).  If 0, a flush will occur every
data collection interval.
.RE

.B -G, --group
Treat process data as an entirely separate group of data, writing it into its own raw file, named
'rawp'.  These separate process files can be played back and processed just like any other 
collectl raw files and in fact one can even play back both at the same time if that is what is
desired.  The only real purpose of this switch is that on some systems with many processes, it is
possible to generate huge raw files (some have been observerd to be >250MB!) and while collectl
will happily play back/process these files it can take a long time.  By using the -G switch one
still gets a hugh rawp file, but the raw file is a much more manageable size and as a result much
faster to process.

.B "-i, --interval interval[:interval2[:interval3]]"
.RS
This is the sampling interval in seconds.  The default is 10 seconds when run
as a daemon and 1 second otherwise.  The process subsystem and slabs (-sY and -sZ)
are sampled at the lower rate of
.BR interval2.
Environmentals (-sE), which only apply to a subset of hardware, are sampled at
.BR interval3.
Both
.BR interval2
and
.BR interval3,
if specified, must be an even multiple of 
.BR interval1.
The daemon default is -i10:60:300 and all other modes are -i1:60:300.  
To sample only processes once every 10 seconds 
use -i:10.
.RE

.B "-r, --rolllogs time[[,days][,minutes]]"
.RS
When selected, 
.BR collectl
runs indefinately (or at least until the system reboots).
The maximum number of raw and/or plot files that will be retained 
(older ones are automatically deleted) is controlled by the
.BR days
field, the default is 7 days.  The
.BR increment
field which is also optional (but is position dependent) specifies the duration of
an individual collection file in minutes the default of which is 1440 or 1 day.
.RE

.B "--quiet"
.RS
Whenever collectl wants to tell the user something, it assigns a category to it such as
Informational, Warning, Error or Fatal.  When run with -m, all messages are displayed 
for the user and if logging data to a file with -f, these messages are also sent to a
log file which is in the data collection directory and has an extenion of "log".  
However, if -m is not specified Informational messages (such as collectl starting
or stopping) are not reported on the terminal but the other 3 are.  Sometimes the 
warnings can be annoying and one can suppress these with --quiet though they will still be
written to the message log in -f.  You cannot suppress Error or Fatal errors.
.RE

.B "--rawtoo"
.RS
Only available in conjunction with -P, this switch causes the creation/logging
of raw data in addition to plottable data.  While this may seem excessive,
keep in mind that unlike plottable data, raw data can be played back with different
switches potentially providing more details.  The overhead to write out this 
additional data is minimal, the only real cost being that of extra disk space.
.RE

.B "-R, --runtime duration"
.RS
Specify the duration of data collection where the duration is a number followed
by one of 
.BR wdhms,
indicating how many weeks, days, hours, minutes or seconds
the collection is to be taken for.
.RE

.B "--sep separator"
.RS
Specify the plot format separator - default is a space.  If this is a numeric field it is 
interpretted as the decimal value of the associated ASCII character code.  Otherwise it
is interpretted as the character itself.  In other words, "--sep :" sets the separator 
character to a colon and "--sep 9" sets it to a horizontal tab.  "--sep 58" would also
set it to a colon.
.RE

.B -S, --ssh
.RS
This is typically used when starting collectl on another system via ssh or
rsh.  It causes collectl to 'watch' for its parent (who started it locally) to
exit at which point it will exit as well.  The reason for this switch is that
when the remote command that started collectl exists, collectl's parent will exit
as well but NOT collectl, unless -S is specified.
.RE

Playback Mode

In this mode, data is read from one or more data files that were
generated in Record Mode

.RE
.B "-b, --begin BeginTime"
.RS
Display data from this time forward.  The format of this
field is [yyyymmdd-]hh:mm:ss.  If the 8 digit date is omitted, the date is
taken from the data file.

.RE
.B "-e, --end EndTime"
.RS
Display data thru this time period.  The format of this
field is [yyyymmdd-]hh:mm:ss.  If the 8 digit date is omitted, the date is
taken from the data file.
.RE

.B "-f, --filename Filename"
.RS
If specified, this is the name of a file or directory 
to write the output to (rather than
the terminal).  See the description for details on the format of this field.
This requires the -P flag as well.
.RE

.B "-p, --playback Filename"
.RS
Read data from the specified 
.BR playback
file(s), noting that one can use wildcards in the filename if
quoted (if playing back multiple files to the terminal you probably
want to include -m to see the filenames as they are processed).
The filename must either end 
in 
.BR raw
or
.BR raw.gz.
As an added feature, since people sometimes automate
the running of this option and don't want to hard code a date, you can 
specify the string YESTERDAY or TODAY and they will be replaced in the
filename string by the appropriate date.

.RE

.B "-T, --timezone hours"
.RS
During playback, sample times are reported in the local time at which they
were recorded.  Since this determination is made at the time the playback
file is opened and not for each record, there may be times when a clock had
changed in the middle of a sample and will not be converted correctly.  When
this happens one may have to play back the samples in pieces and manually set
the time offset with -T.
.RE

Common Switches - both record and playback modes
.RE

.B "-d, --debug debug"
.RS
Control the level of debugging information, not typically used.  For details
see the source code.
.RE

.B -h, --help, -x, --helpext
.RS
Display a standard or extended help message.
.RE

.B --hr, --headerrepeat num
.RS
Sets the number of intervals to display data for before repeating the header.
A value -1 will prevent any headers from being displayed and a value of 0
will cause only a single header to be displayed and never repeated.
.RE

.B -l, --limits limit
.RS
Override one or more default exception limits.  If more than one limit they
must be separated by hyphens.  Current values are:

.B SVC:value
.RS 
Report partition activity with Service times >= 30 msec
.RE

.B IOS:value
.RS 
Report device activity with 10 or more reads or writes per second
.RE

.B LusKBS:value
.RS 
Report client or OSS activity greater than limit.  Only applies to
Client Summary or OSS Detail reporting.  [default=100000]
.RE

.B LusReints:value
.RS 
Report MDS activity with Reint greater than limit.  Only applies
to MDS Summary reporting.  [default=1000]
.RE

.B AND
.RS 
Both the IOS and SCV limits must be reached before a device is reported.  This
is the default value and is only included for completeness.
.RE

.B OR
.RS
Report device activity if either IOS or SVC thresholds are reached.
.RE

.B -L, --lustresvc [c|m|o][:seconds]
.RS
This switch limits which servics lustre checks for and the frequency of those checks.
For more information see the man page collectl-lustre.
.RE

.RE
.B -m, --messages
.RS
Write status to a monthly log file in the same directory as the output file 
(requires -f to be specified as well).  The name of the file will be 
.BR collectl-yyyymm.log
and will track various messages that may get generated during every run of 
.BR collectl.
.RE

.B -N, --nice
.RS
Set priority to a 
.BR nicer
one of 10.

.RE
.B "-o, --options Options"
.RS
These apply to the way output is displayed OR written to a plot file.  They
do not effect the way data is selected for recording.  Most of these switches
work in both record as well as playback mode.  If you're not sure, just
try it.

.B 1
.RS
Data in plotting format should use 1 decimal point of precision as appropriate.
.RE

.B 2
.RS
Data in plotting format should use 2 decimal points of precision as appropriate.
.RE

.B a
.RS
Always append data to an existing plot file.  By default if a plot
file exists, the playback file will be skipped as a way of assuring it is 
associated with a single recorded file.  This switch overrides that mechanism
allowing muliple recorded files to be processed and written to a single plot
file.
.RE

.B A
.RS
When playing back one or more files to the terminal in -M1 mode, append the
Average and Totals.
.RE

.B c
.RS
Always open newly named plot fies in 
.BR create
mode, overwriting any old ones
that may already exists.  If one processes multiple files for the same day in
.BR append
mode multiple times, the same data will be appended to the same file mulitple
times.  This assures a new file is created at the start of the processing.
.RE

.B d
.RS
For use with terminal output and  brief mode.  Preceed each line with a date/time stamp,
the date being in mm/dd format.  This option can also be applied to plot formatit
which will cause the date portion to also be displayed in this format as
opposed to D format.
.RE

.B D
.RS
For use with terminal output and brief mode.  Preceed each line with a date/time 
stamp, the date being in yyyymmdd format.
.RE

.B g
.RS
For use with terminal output and brief mode.   When displaying values of 1G or greater
there is limited precision for 1 digit values.  This options provides a way to display
additional digits for more granularity by substituting a 'g' for the decimal point
rather than the trailing 'G'.
.RE

.B G
.RS
For use with terminal output and brief mode.  This is similar to 'g' but preserves
the trailing 'G' by sacrificing a digit of granularity.
.RE

.B m
.RS
Whenever times are reported in plot format, in the normal 
terminal reporting format at the bginning of each interval or when when one 
of the time reporting options (d, D, T or U is selected), append the milliseconds
to the time.
.RE

.B n
.RS
Where appropriate, data such as disk KBs or transfers are normalized to units per 
second by taking the change in a counter and dividing by the number of seconds in 
that interval.  Normalization can be disabled via this option, the result being 
the reported values are not divided by the duration of the interval.
.RE

.B s
.RS
When reporting detailed slab data, leave out slabs with no allocations.
.RE

.B S
.RS
When reporting any slab data, leave out slabs with no activity during the 
current interval.  In other words, only show slabs that change.  Note that
changes in active objects or allocations are not included in this condition
as they change too frequently and do not effect memory allocated for the slabs.
.RE

.B t
.RS
Always start the display for the current interval at the top of the screen
(non-plot format only).  This generates the illusion of a real-time display
when the data fits on a single screen.
.RE

.B T
.RS
For use with terminal output and brief mode, preceeds each line with a time stamp.
.RE

.B u
.RS
Create plot files with unique names by include the starting time of a colletion
in the name.  This forces
multiple collections taken the same day to be written to multiple files.
.RE

.B "u or --utc"
.RS
In plot format only, report timestamps in Coordinated Universal time which is more
commonly know as UTC.
.RE

.B x
.RS
Report only exception records for selected subsystems.  Exception reporting also requires
--verbose.  Currently this only 
applies to disk detail and Lustre server information so one must select at least 
-s D, l or L for
this to apply.  If writing to a detail file, this data will go into a separate
file with the extension 
.BR X
appended to the regular detail file name.
.RE

.B X
.RS
Report both exceptions as well as all details for selected subsystems, for
-s D, l or L only.
.RE

.B z
.RS
If the compression library has been installed, all output files will be compressed by
default.  This switch tells collectl not to compress any plottable files.  If collectl
tries to compress but cannot because the library hasn't been installed, it will generate
a warning which can be suppressed with this switch.
.RE
.RE

.RE
.B "-O, --subopts Sub-system Options"
.RS
These options apply to specific subsystems as opposed to 
.BR -o
which apply to all subsystems.  Some control which data is to be collected and
others may control which data is displayed.

.B 2
.RS
Collect nfs V2 data
.RE

.B 3
.RS
Collect nfs V3 data
.RE

.B B
.RS
Display Lustre OST I/O distribution by buffersize, where the buffers range in size from 1 to 
128 pages and the size of a page is installation dependent.  All IA32 systems have a page size of 4K.
.RE

.B C
.RS
Collect nfs statistics for the CLIENT rather than the SERVER.
.RE

.B D
.RS
For lustre MDSs and OSTs, collectl disk block iostats.
.RE

.B M
.RS
For lustre clients, collect metadata.
.RE

.B R
.RS
For lustre client, collect readahead stats
.RE
.RE

.RE
.B -P, --plot
.RS
Generate output in plot format.  This format is space separated data which 
consists of a header (prefaced with a # for easy identification by an analysis
program as well as identifying it as a comment for programs, such as gnuplot,
which honor that convention).  When written to disk, which is the typical way
this option is used, 
.BR summary
data elements
are written to the 
.BR tab
file and the 
.BR detail
elements written to one or
more files, one per detail subsystem.  
If -f is not specified, all output is sent to the terminal.  
Output is always one line per sampling interval.
.RE

.B "-s, --subsys subsystem"
.RS
This field controls which subsystem data is to be collected or played back
for. The rules for displaying results vary depending on the type of data to be
displayed.  If you write data for CPUs and DISKs to a raw file and play it back
with -sc, you will only see CPU data.  If you play it back with -scm you will
still only see CPU data since memory data was not collected.  However, when 
used with -P, collectl will always honor the subsystems specified with 
this switch so in the previous example you will see CPU
data plus memory data of all 0s.  To see the current set of default subsystems,
which are a subset of this full list,
use -h.

You can also use + or - to add or subtract subsystems to/from the default values. 
For example, '-s-cdn+N'< will remove cpu, disk and network monitoring from the
defaults while adding network detail.

The default is 'cdn', which stands for CPU, Disk and Network data.

SUMMARY SUBSYSTEMS

.B "c - CPU"

.B "d - Disk"

.B "f - NFS V3 Data"

.B "i - Inode and File System"

.B "j - Interrupts"

.B "l - Lustre"

.B "m - Memory"

.B "n - Networks"

.B "s - Sockets"

.B "t - TCP"

.B "x - Interconnect"

.B "y - Slabs (system object caches)"

DETAIL SUBSYSTEMS

This is the set of 
.BR detail
data from which in most cases the corresponding summary data is
derived.  There are currently 2 types that do not have corresponding summary
data and those are 'Environmental' and 'Process' (in fact, 'Process' has its own
manpage named 'collectl-process').  So, if one has 3 disks
and chooses 
.B -sd,
one will only see a single total taken
across all 3 disks.  If one
chooses 
.B -sD,
individual disk totals will be reported but no totals.  Choosing 
.B -sdD
will get you both.

.B "C - CPU"

.B "D - Disk"

.B "E - Environmental data (fan, power, temp)"

.B "F - NFS Data"

.B "J - Interrupts"

.B "L - Lustre OST detail OR client Filesystem detail"

.B "LL - Lustre client OST detail.  LL overrides L"

.B "N - Networks"

.B "T - 65 TCP counters only available in plot format"

.B "X - Interconnect"

.B "Y - Slabs (system object caches)"

.B "Z - Processes"
.RE

.B --showheader
.RS
In collectl mode this command will cause the header that is normally written to a data file to
be displayed on the terminal and collectl then exists.  This can be a handy way to get a brief
overview of the system configuration.
.RE

.B --showoptions
.RS
This command shows only the portion of the help text that desribes the -o and --options switches
to save the time of wading through the entire help screen.
.RE

.B --showrootslabs
.RS
This command only works on systems using the new slab allocator and will list the root 
name (these are those entries in /sys/slab which are not soft links) along with all 
its alias names.  If a name doesn't have an alias, it will not appear in this report.
.RE

.B --showslabaliases
.RS
This command only works on systems using the new slab allocator.  Like --showrootslabs, it
will name a slab and all its aliases but rather than show the root slab name 
it will show one of the aliases to provide a more meaningful name.  If there are any 
slabs that only have a single (or no) alias they will not be included in this report.
.RE

.B --showsubopts
.RS
Similar to --showoptions, this command summaries just the paramaters associated with -O and
--subopts.
.RE

.B --showsubsys
.RS
Yet another way to summare a portion of the help text, this command only shows valid subsystems.
.RE

.B "--top [num]"
.RS
Include the top consumers of total cpu for this interval.  In interactive mode and if not specified, the process
monitoring interval will be set to that for other subsystems.  The screen will be cleared for each interval
resulting in a display similar to the 'top' utility.  In playback more the screen will NOT be cleared.  You
cannot use this switch in 'record' mode.
.RE

.B -v
.RS
Show version and whether or not Compression and/or HiResTime modules have
been installed and exit.
.RE

.B -V
.RS
Show default parmeter and control settings, all of which can be changed in
/etc/collectl.conf
.RE

.B --verbose
.RS
Display output in verbose mode.  This often displays more data than in the default mode.  When 
displaying detail data, verbose mode is forced.  Furthermore, if summary data for a single 
subsystem is to be displayed in verbose mode, the headers are only repeated occasionally whereas
if multiple subsystems are involved each needs their own header.
.RE

.B -w
.RS
Disply data in
.BR wide
mode.  When displaying data on the terminal, some data is formatted followed 
by a K, M or G as appropriate.  Selecting this switch will cause the 
full field to be displayed.  Note that there is no attempt 
to align data with the column headings in this mode.
.RE

.SH DESCRIPTION

The
.BR collectl
utility is a system monitoring tool that records or displays
specific operating system data for one or more sets of subsystems. Any set
of the subsystems, such as CPU, Disks, Memory or Sockets can
be included in or excluded from data collection.  Data can either be
displayed back to the terminal, or stored in either a compressed or
uncompressed data file. The data files themselves can either be in 
.BR raw
format
(essentially a direct copy from the associated /proc structures) or in a space
separated 
.BR plottable
format such that it can be easily plotted using tools 
such as gnuplot or excel.  Data files can be read and manipulated from the
command line, or through use of command scripts.

Upon startup,
.BR collectl.conf
is read, which sets a number of default parameters and switch values.  Collectl
searches for this file first in /etc, then in the directory the collectl
execuable lives in (typically /usr/sbin) and finally the current directory.
These locations can be overriden with the 
.BR -C 
switch.  Unless
you're doing something really special, this file need never be touched, the
only exception perhaps being when choosing to run collectl as a service and you
wish to change it's default behavior which is set by the DaemonCommand entry.

.SH RESTRICTIONS/PROBLEMS

Thread reporting currently only works with 2.6 kernels.

The pagesize has been hardcoded for perl 5.6 systems to 4096 for IA32
and 16384 for all others.  If you are running 5.6 on a system with a
different pagesize you will see incorrect SLAB allocation sizes and will
need to scale the numbers you're seeing accordingly.

I have recently discovered there is a bug in /proc in that an extra line
is occasionally read with the end of the previous buffer!  When this
occurs a message is written (if -m enabled) and always written to the
terminal.  Since this happens with a higher frequency with process data
I silently ignore those as the output can get pretty noisey.  
If for any reason this is a problem, be sure to let me know.

Since collectl has no control over the frequency at which data gets written
to /proc, one can get anomolous statistics as collectl is only 
reporting a snapshot of what is being recorded.  For more information
see http://collectl.sourceforge.net/TheMath.html.

At least one network card occasionally generates erroneous network stats and to try
to keep the data rational, collectl tries to detect this and when it does generates
a message that bogus data has been detected.

.SH FILES, EXAMPLES AND MORE INFORMATION

http://collectl.sourceforge.net OR /opt/hp/collectl/docs

.SH ACKNOWLEDGEMENTS
I would like to thank Rob Urban for his creation of the Tru64 Unix
collect tool, which collectl is based on.

.SH AUTHOR
This program was written by Mark Seger (Mark.Seger@hp.com).
.br
Copyright 2003-2008 Hewlett-Packard Development Company, LP
.br
collectl may be copied only under the terms of either the Artistic License
or the GNU General Public License, which may be found in the source kit

.SH SEE ALSO
